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frame basis. Since resources are finite, processing may not be completed for a given level 
of output quality by the deadline for the completion of this output processing, i.e., a 
deadline miss occurs. Each deadline miss results in severe artifacts in the output. Due to 
the wireless media nature, the number of layers received for a given frame varies over 
5 time, which restricts the number of quality levels that can be chosen for the frame. In a 
preferred embodiment, every time a media frame is received, e.g., a video frame, and 
must be displayed, the number of received layers is inspected. The maximum number of 
layers that can be processed is determined by the number of received layers for a frame 
and the time that the CPU is available to process the layers of that frame with minimal 
10 risk of missing the corresponding deadline. However, quality level changes may result in 
perceivable artifacts. By minimizing the number of quality level jumps over time, i.e., 
smoothing the received images over time, the user views an image having a fairly stable 
quality. This smoothing is done, in a preferred embodiment, by setting up a Markov 
chain and defining a value function. Quality level changes that are not caused by the 
network conditions yield much negative value. Quality level changes that are caused by 
network fluctuations yield zero value in the case of quality drop. Showing no image at all 
receives the highest penalty. On the other hand, a higher number of processed layers 

yields a higher value. 

By playing many videos with realistic packet losses a layer selection procedure 
has been developed that is optimized with respect to the value function. The optimized 
layer selection function developed in this manner, is used to determine the number of 
layers that need to be displayed as a function of the number of received layers for a given 
frame and for the preceding frames. 

There are a number of approaches that deal with optimizing resource consumption 
25 and maximizing output quality. One is the approach of the present invention to apply 
scalable video algorithms to the decoding of scalable video. A quality level is defined as 
a number of layers to be processed. Prior-art algorithms assume a stable input (like 
DVD). Stable input means that there is no loss of information during transmission, thus it 
implies that during decoding of the video-data^ny quality level can be chosen. The 
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internal setting for resolving a trade-off between resource usage and output quality. FIG. 
1 illustrates the basic concept of a scalable video processor with a control mechanism 102 
influencing the behavior of a scalable application 101 by means of a set of parameters 
103. The use of scalable applications to accomplish video processing allows parts of the 

5 application to be readily scaled so that output qualities can be achieved thereby enabling 
resource consumption to be balanced against output quality. 

Consider a video decoder as a scalable video application (SVA). This video 
decoder can be controlled by varying its internal settings to produce an output video 
stream of variable quality. As illustrated in Table 1, the decoder processes only the base 

10 layer when it operates at the lowest quality level. With the increase of the quality level, 
the decoder increases the number of layers to be processed, as well as the processing time 
(and, obviously the resource consumption). 



Quality level 


Number Of Layers To Be Processed 


qo 


BL 


qi 


BL+ELi 


qz 


BL+EL1+EL2 


... 


... 




BL+EL1+EL2+. . .+EL N 



15 TABLE 1 



However, not all the layers are always available. Given that the decoder receives 
the layers from a network, there is no guarantee for the number of layers input to the 
decoder at any moment in time. Therefore, it is uncertain what number of layers will be 
20 processed next. Information about the number of available layers can be obtained from 
the input buffer. 

In general the processing by the scalable application of the present invention is 
described as follows. The application fetches a unit of work (frame) from an input 
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minimizes both the number of deadline misses and the number of quality level changes, 
while maximizing the quality level. 

According to the invention, this problem is modeled as a Markov decision 
problem. The model is based on calculating relative progress of an application at its 
5 milestones. Solving the Markov decision problem results in a quality level control 
strategy that can be applied during run time while incurring little overhead. 

Consumer terminals, such as set-top boxes and digital TV-sets, are required by the 
market to become open and flexible. This is achieved by replacing several dedicated 
hardware components, performing specific media processing applications, by a central 
10 processing unit (CPU) on which equivalent media processing applications execute. 
Resources, such as CPU time, memory, and bus bandwidth, are shared between these 
applications. Here, preferably only the CPU resource is considered. 

At each milestone, the relative progress of the application is calculated. Here, the 
relative progress at a milestone is defined as the time until the deadline of the milestone, 
1 5 expressed in deadline periods. 

Relative progress at milestones can be calculated as follows. Assume, without 
loss of generality, that the application starts processing at time t=0. The time of milestone 
m is denoted by tw Next, the deadline of milestone m is denoted by d m . The deadlines 
are strictly periodic, which means that they can be written as 



20 



d m = do + m*P 

where P is the period between two successive deadlines and do is an offset. The relative 
progress at milestone m, denoted by p™, is now given by 

To illustrate the calculation of relative progress, consider the example timeline 
shown in FIG. 2. In this example, P=l and do=l • The relative progress at milestones 1 up 
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progress intervals 

,fork=0 firh 

5 

The lower bound and the upper bound of a progress interval tt is denoted by n_ and n , 
respectively. 

At each milestone, a decision must be taken about the quality level at which the 
next unit of work will be processed. Hence, the set of decisions that can be taken in a 
10 state, i.e., in the Markov decision problem, corresponds to the set of quality levels at 
which the application can run. This set is denoted by Q. 

Every quality level corresponds to the number of layers ,that are processed. 
Therefore, it is not possible to choose the quality level which requires decoding more 
layers than there are in the input buffer for a given frame. Thus the maximal quality level 
15 that can be chosen is given by the number of layers received and is defined by maxq(i). 

Quality level changes are also taken into account, thus at each milestone the 
previously used quality level must be known. This can be realized by extending the set of 
states with quality levels. Therefore, a state i is defined by 

• the relative progress interval in state i, denoted by 7i(i); 

20 • the maximal quality level that it is possible to choose for the next unit of 

work in state i, denoted by maxq(i); 

• the previously used quality level in state i, denoted by q(i). 
Therefore, the set of states becomes II x Q x Q. 

A second element of which Markov decision problems consist is a set of 
25 transition probabilities. Let p% denote the transition probability for making a transition 
fiom a state i at the current milestone to a state j at the next milestone, if quality level q is 
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Then it is derived: 



Y 



Pfe* A.+, <xhP(p».x >*)-P{p m +* **) otherwise. 



10 



Let F, denote the cumulative distribution function of X, and make a pessimistic 
approximation of p m by choosing the lowest value in the interval 



Given the above, the probabilities p" 9 can be approximated by 



(3) 



l aaxq m jmaq Mt , 



maxq„^noxq mtl 



Y *. 

nuuq m jmixq mt , ? 



*F g (b{l-n(j)+*(t))) 
F (&(1-20)+S(0))-^,(6(1-^0')+S 



otherwise. 



15 The more progress intervals are chosen, the more accurate the modeling of the 

transition probabilities is, as the approximation in (3) is better. 

A third element of which Markov decision problems consist is revenues. The 
' revenue for choosing quality level q in state i is denoted by r f ». Revenues are used to 
implement the three problem objectives. 
20 First, the quality level at which the units of work are processed should be as high 

as possible. This is realized by assigning a reward to each rf , which is given by a 
function u(q). This function is referred to as the utility function. It returns a positive 
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is shown in FIG. 4. This FIG. illustrates that, for example, if the relative progress at a 
particular milestone is equal to 1, and if the previously used quality level is q 3 and the 
maximum quality level for the next frame is q 3 , then quality level q 3 should be chosen to 
5 process the next unit of work, i.e., the next frame. 

Without loss of optimality, so-called monotonic control strategies can be used, 
i.e., per previously used quality level it can be assumed that a higher relative progress 
results in a higher or equal quality level choice. Then, for storing an optimal control 
strategy, per previously used quality level only the relative progress bounds at which the 
10 control strategy changes from a particular quality level to another one have to be stored. 
A control strategy therefore has a space complexity of 0(|Qf), which is independent of 
the number of progress intervals. 

The Markov decision problem can be solved off-line, before the application starts 
executing. Next, we apply the resulting control strategy on-line, as follows. At each 
15 milestone, the previously used quality and the maximum quality levels are known, and 
the relative progress of the application is calculated. Then, the quality level at which the 
next unit of work is to be processed is looked up. This approach incurs little overhead. 

As input for the experiments anMPEG-2 Signal to Noise Ratio (SNR) decoding 
trace file of a video sequence consisting of 120000 frames is used. This trace file 
20 contains for each frame the processing time required to decode it, expressed in CPU 
cycles on a TriMedia, in each of four different quality levels, labeled q 0 up to qa in 
increasing quality level order. That is, the number of enhancement layers was set to 3 
and the bit-rate for all layers is equal. 

As a first step in the evaluation of the present invention, an assumption was made 
25 that the probabilities of transition from one maximal quality level to another are equal. 
Therefore, at milestone m the variable 

has the same value for any pair maxqm and maxq^i . 
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The first test used a budget of 40 ms and maximum quality level, assuming that 
network throughput is sufficient for delivery of all layers. Table 2 contains the changes 
in quality levels for the scalable application of the present invention. Table 3 contains the 
changes in quality levels for the straightforward application. As shown in Table 2 and 
Table 3, the straightforward algorithm makes a change in the quality level on average 
every 4 th frame, which is 1300 times the number of changes made by the present 
invention. At the same time, the average quality for the scalable application of the 
present invention is higher than for the straightforward application, as illustrated in Table 
4, which illustrates the percentage of quality level usage. 
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TABLE 2 - Changes of Quality Levels for the Scalable Application in Test 1 
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TABLE 5 - Changes of Quality Levels for the Scalable Application in Test 2 
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TABLE 6 - Changes of Quality Levels for the Straightforward Application in 
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1 . quality level is maximized; 

2. deadline misses are minimized; and 

3. quality level changes are minimized, 
taking into account the maximum quality level determined by the number of received 
layers. Post processing was not taken into consideration. 

Restating these objectives, a quality level control strategy was developed for a 
scalable media processing application, which had been allocated a fixed CPU budget 
such that it nunimized both the number of deadline misses and the number of quality 
level changes, while maximizing the quality level. A parameter in the model is the 
number of quality level changes, the fewer the number of changes the smoother the 
image viewed. 

FIG. 8 illustrates a system 1200 according to the invention in a schematic way. 
The system 1200 comprises memory 1202 that communicates with the central processing 
unit 1210 via software bus 1208. Memory 1202 comprises computer readable code 1204 
designed to determine the amount of CPU cycles to be used for processing a media frame 
as previously described. Further, memory 1202 comprises computer readable code 1206 
designed to control the quality level of the media frame based on relative progress of the 
media processing application calculated at a milestone. Preferably, the quality level of 
processing the media frame is set based upon a Markov decision problem that is modeled 
for processing a number of media frames as previously described. The computer readable 
code can be updated from a storage device 1212 that comprises a computer program 
product designed to perform the method according to the invention. The storage device is 
read by a suitable reading device, for example a CD reader 1214 that is connected to the 
system 1200. The system can be realized in both hardware and software or any. other 
25 standard architecture able to operate software. 

FIG. 9 illustrates a television set 1310 according to the invention in a schematic 
way that comprises an embodiment of the system according to the invention. Here, an 
antenna, 1300 receives a television signal. Any device able to receive or reproduce a 
television signal like, for example, a satellite dish, cable, storage device, internet, or 
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CLAIMS: 

1. A method of setting a quality level of an output image of a media frame by a 
media piocessing application, comprising the steps of: 

detennining an amount of resources to be used for processing the media frame; 
controlling the quality level of the output image based on 

1. relative progress of the media processing application calculated at a milestone, 

ii. a maximal quality level that is possible to choose for the output image, 

iii. a previously used quality level of an output image, and 

iv. a maximum quality level based on the number of received layers 

2. The method of claim 1, wherein the quality level is chosen based on a minimum 
of the highest quality level possible for processing the next frame and a highest quality level 
required to maintain the quality of the output image. 

3. The method of claim 1, wherein: 

the step of controlling the quality level of the media frame is modeled as a Markov 
decision problem comprising a set of states, a set of decisions, a set of transition probabilities and 
a set of revenues; 

solving the Markov decision problem to derive an optimal strategy, and 
determining the number of layers of the media frame that are decoded based upon this 
solution. 

4. The method of claim 3, further comprising the steps of: 

defining the set of states to comprise the relative progress of the media processing 
application at a milestone and the previously used quality level; 

defining the set of decisions to comprise a plurality of quality levels that the media 
processing application can provide; 
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8. The system of claim 7, wherein: 

the set of states comprises the relative progress of the media processing application at a 
milestone and a previously used quality level of a previous media frame; 

the set of decisions comprises a plurality of quality levels that the scalable media 
processing application can provide; 

the set of transition probabilities comprises a probability that a transition is made from a 
state of the set of states at a current milestone to another state of the set of states at a next 
milestone for a given quality level of the plurality of qualities; and 

the set of revenues comprises a positive revenue related to a positive quality level of the 
media frame, a negative revenue related to a deadline miss and a negative revenue related to a 
quality level change. 

9. A computer program product designed to perform the method according to claim 



1. 



10. 



A storage device comprising a computer program product according to claim 9. 



11. 



A television set comprising a system according claim 5. 



12. 



A set-top box comprising a system according claim 5. 
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